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Today’s Presentation 



• Illustrates some key strategic aspects of conducting 
effective concept design & design-to-cost trade studies 

> What concept design is & why it’s important 

> Fidelity needed in concept design solution 

> Techniques in designing mission level trade space 

> Challenges in determining credible design convergence 

> Recommended practices 
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Important Note 



• Concept design may be conducted using variety of methods 

> This presentation describes selected aspects of one method for 
conducting a concept design study 

□ Uses a space observatory example 

> Method best suited to immature mission concepts that advance state 
of the art or that have high design uncertainty 
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What Concept Design is 


Why it’s Important 
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Concept Design is Exploratory Process to 
Determine System Level Design Baseline 


• Conducted in pre-Phase A & Phase A of Project Life Cycle to 
provide “feasible” system design baseline for new concept 


• As much an investigation of requirements as of design 


> Concurrent investigation of: 

□ Concept of operations 

□ Requirements 

□ Design 

□ Performance 

□ Technology development 

□ Verification approach 

□ Flight dynamics 

□ Ground segment (ground stations, mission & science ops centers) 

□ Launch interface 

□ Cost 

□ Schedule 

□ Risks, etc. 
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Concept Design Occurs in Pre-Phase A & Phase A 

of Project Life Cycle 



NASA Project Life Cycle 

NASA Procedural Requirements 7120.5E 



Figure 1 
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Preliminary Design Occurs in Phase B 
Detailed Design Occurs in Phase C 



Concept Design Plays Central Role in 
Project Success 



• Earliest life cycle phases have most leverage over life cycle cost 
(LCC) 

> Concept design product effectively locks (or renders unchangeable) 
-70% of system LCC 
□ Per ref. (a) & ref. (b) 


• Such extraordinary leverage presents business case for 
conducting concept design in pragmatic & rigorous fashion 

> Particularly important for immature mission concepts that advance 
state of the art or that have high design uncertainty 
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Concept Design Plays Central Role in 
Project Success (Cont’d) 



• Done well, provides executable system level design baseline 
for project teams in Phase B & later phases 

• Not done well, can subject project teams in Phase B & later 
phases to system level redesign - in some cases, to multiple 
system level redesigns accompanied by: 

> Fluid technical baselines with ever-decreasing capabilities 

> Cost overruns & recurring schedule delays 

> Contract disputes & cancellations 

> Challenges in retaining trained personnel 
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P re- Phase A / Phase A Offer Unique Venue 
for System Level Trades 


• Teams small, agile, closely coordinated 

> Typically operate absent many formalities of later project phases 

□ e.g., typically no prime contracts, system level requirements not under 
configuration control until late in phase A 

> Can accommodate high rate of change in system level 
“requirements” & design characteristics (R&DC) 

□ Enables broad investigation of trade space in relatively short time 

• Note: 

> “ requirements ” in quotes denotes interim reference capabilities used 
to guide evaluation of point designs in trade space 

> System level requirements aren't baselined until SRR for a final 
concept design that meets technical & programmatic (including cost 
& schedule) constraints 
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Phase B & Later Development Phases Not 
Well Suited for System Level Trades 


• In Phase B, system level design is more difficult & expensive 
to change , e.g., 

> Teams typically larger & more distributed 

> Prime contracts typically in place 

> System level requirements typically under configuration control 

> Preliminary design work assumes system level design complete 

• In Phases C & D, system level changes even more difficult & 
expensive to change 

> Teams typically even larger than in Phase B 

> System & subsystem level requirements typically under configuration 

control 

> Detailed design work either underway or has been completed 
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Fidelity Needed in Concept Design 

Solution 
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A Proposed Definition for “ Feasible ” 



• The term “feasible” is used frequently in concept design , but 
its use is often problematic 

> Often left undefined & subject to interpretation 

• This presentation uses “feasible” mission concept to mean: 

> Technical, cost, & schedule characteristics for a single, baseline 
mission concept design have been credibly converged to the 1st 
order by the end of Phase A, 

> such that the design may be developed, launched, operated, & 
decommissioned by a competent project team starting in Phase B 
within customary technical & programmatic margins 
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A Proposed Metric for Level of 
Convergence (1 of 2) 


• Credible convergence to 1 st order by end of Phase A means: 

> System level sizing & performance (SLSP) of mission elements is 
confidently determined to within 90% of SLSP when flight system is 

delivered 

□ For given cost & schedule constraints 

> i.e., there is residual uncertainty that SLSP could change by ± -10% 
between end of Phase A & launch 
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A Proposed Model for Product Fidelity 
During Design Phases (Solid Black Curve)* 


Figure 2 


* Adapted from ref. (c). Fig. 3-4 
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A Proposed Metric for Level of 
Convergence (2 of 2) 



• Solid black curve in Fig . 2 (uniform convergence) shows 
allowable SLSP error decreases as design moves from 
Phases Pre-A through C 

> End Phase A: 1 st order, or 90% (accurate to 1 digit, ~ ± 10% error)* 

> End Phase B: 2 nd order, or 99%> (accurate to 2 digits, ~ ± 7 % error) 

> End Phase C: 3 rd order, or 99.9% (accurate to 3 digits, ~ ±0.1%) error) 

• Metrics for SLSP error are approximate guidelines only 

> Coarse model that depicts an idealized trend of fidelity in each phase 

> Assume calculations done properly, but with incomplete or incorrect 
information / assumptions 

• * read as 9 x 10 1 %, accurate to 1 significant digit 
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Example SLSP Error Convergence for 
Mass 


• For a 4,000 kg space observatory, system level mass should 
be known to: 

□ End Phase A: Within ~ ± 10%, or ~ ± 400 kg of final launch mass 

□ End Phase B: Within ~ ± 1%, or ~ ± 40 kg of final launch mass 

□ End Phase C: Within ~ ±0.1%, or ~ ± 4 kg of final launch mass 
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Role of (Selected) Resource Margins on 
Required Convergence 



• Solid black curve in Fig . 2 must be within allowable margins 

> Power & Dry Mass Margin requirements (per ref. (d)) are shown in 

Fig. 2 

□ End Phase A: >25% 

□ End Phase B: > 20% 

□ End Phase C: > 15%> 

• Cost (not shown in Fig. 2) serves as design constraint 

> Cost margin (per ref. (e)) 

□ Cost through Phase D: > 30%) (guideline at Phase B start) 

□ Cost through Phase D: > 25%> (requirement at Phase C start) 

• Other programmatic margin requirements apply as well, e.g., 

> Schedule margin (per ref. (e)), not shown in Fig. 2 
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Importance of Concept Design 
Convergence to Project Manager 



• Project Manager at start of Phase B holds 25% margins for 
power & dry mass resources (Fig. 2) 

> Can accgmijigdate concept design credibly converged to within 10% 
of flight sizing & performance values for power & dry mass 

□ Even if 10% error occurs in direction of needing more resources 

> CatYt accgmijigdate concept design credibly converged to within 
30% of flight sizing & performance values for power & dry mass 

□ if 30% error occurs in direction of needing more resources 

□ Design de-scope likely required 
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Techniques in Designing Mission Level 

Trade Space 
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Goal: Maximize Technical Capability within Cost & 
Schedule Constraints (Solid Red Lines) 



Concept Design Mission Level Trade Space 

Selecting Trades to Expedite Convergence - 3 Cycle Example 

Figure 3 
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Concept Design Mission Level Trade Space 

Selecting Trades to Expedite Convergence - 3 Cycle Example 
(Cont’d) 


• Approach in Fig. 3 deduces R&DC for C design by interpola- 
ting on results from A & B designs (bounding cases) 

> Technical capability of point C isn’t known at outset of study 

• More like root finding algorithm than like successive 
refinement design process typically used in Phases B & C 

> In Phases B & C, each design is refinement of “baseline” system 
level design from prior phase 

> In concept design process discussed here, typically there isn’t a 
“baseline” system level design until concept design is complete 

• Purposely views design problem from multiple perspectives 

> Illuminates aspects that otherwise may have remained hidden 

□ Helps stimulate creative thinking & mitigate biases 

□ Helps discover “unknown unknowns” (UUs) 
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Why Selecting Bounding Cases is 
Important 


• Failure to select bounding cases may cause extrapolation to 
determine R&DC for final solution 

> Adds risk in technical, cost, & schedule estimates 

> May result if both A & B designs exceed cost & schedule constraints 

□ Implies R&DC for B design didn’t identify “true” science or technology floor 
(presumes a solution exists) 

• Or, may cause need for more design cycles 

> Deadline may not permit, or may drive significant team overtime 

• Optimistic A designs & “false” science floors for B designs 
are common 

> Customer’s vision often isn’t cost / schedule constrained 

> Customer may resist identifying “true” science or technology floor 

• Teams that recognize, or adapt to, these considerations 
pragmatically & quickly fare better than teams that don’t 
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Selecting R&DC (Typical Case) 



• Typical Approach 

> A Design: Most * parameters reflect realistic desired capability 

> B Design: Most * parameters reflect science or technology floor 

> C Design: Most * parameters are between A & B capabilities 

* but not necessarily all 


• R&DC for B design reevaluated after A design to assure 
solution space bounded 

> Presumes A design done first 

• Many parameters varied concurrently due to need to cover 
broad solution space in limited time** 

> Experience shows teams can sufficiently understand parameter 
sensitivities 

** after approach originally used by Mr. John Oberright, NASA / GSFC Emeritus, for 
Space Technology-5 concept design study (1999) 
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Challenges in Determining Credible 
Design Convergence 
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Convergence Indicators Difficult to Define 
Objectively 



• Concept design is inherently an exploratory process with 
relatively high uncertainty 

• Concept design teams learn at high rate 

> Early assumptions & conclusions may be invalidated by later findings 
or by unpredictable discovery of UUs 
□ Convergence can appear non-uniform (see copper line in Fig. 2) 


• Yet, indicators are desired to help avoid inferring 
convergence prematurely, e.g., due to: 

> Insufficient rigor 

> Study funds or time being exhausted 

> Pressure to meet a milestone deliverable, etc. 

> Biases 
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Convergence Determinations Often 
Evident Only in Hindsight 

Figure 4 
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Cost (or cost & schedule) estimates typically grow for 
designs A & B after accounting for knowledge gained in 
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Why Early Cost Estimates Tend to be 
Optimistic 



• A common characteristic of concept design is costs for a 
given design tend to increase with each design cycle 

> Particularly true for immature mission concepts that advance state of 
the art or that have high design uncertainty 


• As teams progress through cycles , they learn more of what 
may have been omitted / incorrectly assumed in prior cycles 

> After B cycle , cost of A design may increase 

> After C cycle , cost of A design may increase again , & cost of B 
design may increase 

□ Causes A & B points to move to right in Fig. 4 

> When accompanied by schedule increases, A & B points also move 
into page 


> After C cycle, learning tapers off for most designs 

□ Occasionally, a D cycle is needed (or may be planned from outset) 



D.A. Di Pietro 
Goddard Space Flight Center 


Published and used by NDIA with permission 



Why Early Cost Estimates Tend to be 
Optimistic (Coni’ d) 


• Cost analysis is normally performed using multiple methods 

> One method is “grass roots” - uses relatively detailed work 
breakdown structure (WBS) 

• WBS dictionary for most space mission elements is 
relatively well known & largely existing, e.g., 

> Spacecraft, launch, ground systems, etc. 

• Conversely, WBS dictionary for new instruments is unique 

> Design dependent, evolves as instrument design evolves 

> Key aspect for designs dominated by new instruments 


• Multiple cost cycles typically needed to develop well 
understood WBS free of significant gaps & overlaps 



> Cost fidelity improves with understanding of design and WBS 

> Gaps common in design & cost in early cycles as team learns 
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Subjective Criterion for Convergence 
Determination - Significant Surprises 



• One subjective criterion for credible convergence is whether 
team has experienced significant surprises 

• Team that hasn’t experienced at least a few significant 
surprises should be cautious of its results 

• Lack of surprises may indicate: 

> Team hasn’t progressed sufficiently down learning curve 

> Team didn’t sufficiently exercise trade space or mitigate biases 

> Concept design study objective wasn’t sufficiently challenging 
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Recommended Practices 
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General Guidance 


Treat design cycles as precious resource 

> Essential, but in limited supply due to time & resources available 

Don't retrofit A & B designs with insights from later cycles 

> Time better spent just applying learning to final design 

Document design results in reports (not briefings) at end of 
each cycle (see rationale in backup charts) 

> Reports (functional, not pristine) are record documents 

> Briefings, if needed, are built exclusively from approved reports 

Focus on what ee should ” be done vs. what “can” be done 

> Address 1 st order items that demand attention early 

□ Defer lower order items to later phases 

> Focus team efforts on developing product, omit peripheral tasks 
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Analogy for 1 st Order Level of Analysis 
Depth in Concept Design 


• Pre-Phase A & Phase A teams evaluate multiple designs in 
broad trade space in relatively short period 

> Analysis tools used typically are 1st order precision, agile enough to 
adapt to frequent /significant system level changes 
□ Analogy: “ Hacksaw ” 


• By comparison, analysis tools typically used in: 

> Phase B are 2 nd order precision, assume system level design stable 
□ Analogy: “File” 


> Phase C are 3 rd order precision, assume both system & subsystem 
level designs stable 
□ Analogy: “ Polisher ” 
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Analogy for 1 st Order Level of Analysis 
Depth in Concept Design (Cont’d) 


• Team using “hacksaw” in Phase C has done something 
wrong 

> Didn’t credibly converge 1 st order solution by end of Phase A 

> Re-doing concept design work late & out of sequence 

• Team using “polisher” in Phase A is doing something wrong 

> Won’t move quickly or broadly enough to rough-out & credibly 
converge 1 st order solution * 

□ Recognize some design elements may not even exist in final concept 

design 


Some high risk elements may selectively warrant added scrutiny 
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Avoid Significant Rounding Errors 



• To avoid masking resource margins , bookkeep design & 
performance calculations to 3 significant digits & report out 
to 2 significant digits 

> Should not be taken to imply there is 3-digit accuracy in concept 
design work - there usually is not 

> Simply a numerical safeguard to avoid propagating rounding errors 
that could overwhelm ability to adequately determine design or 
performance margins 

> See margin example in backup charts 
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Recognize Typical (but Unofficial) Phases 
of Concept Design 


• Concept design teams developing immature mission 
concepts that advance state of the art often experience four 
phases of work 

> 1) Unbridled Optimism 

> 2) Shock 

> 3) Denial 

> 4) Acceptance 

• The quicker a team moves through phases 1,2, & 3 and 
arrives at Phase 4, the better that team will fare 

> See backup charts for additional discussion 


D.A. Di Pietro 
Goddard Space Flight Center 


Published and used by NDIA with permission 


35 



Closing Thoughts 



D.A. Di Pietro 
Goddard Space Flight Center 


Published and used by NDIA with permission 


36 


Closing Thoughts (1 of 2) 



• Concept design phases have extraordinary leverage over 
project success, it’s important they be: 

> Conducted in rigorous & pragmatic fashion 

□ Particularly for immature mission concepts that advance state of the art or 
that have high design uncertainty 

> Credibly converged to 1 st order prior to Phase B 

□ Project Manager relies upon this 

• Unknowns dominate for designs that advance state of art 

> Be cautious of early results, they may not be as initially appear 

> Use bounding trades to help discover major UUs & mitigate biases 

> Look for evidence of significant surprises / unexpected findings 

□ Indicate team progressing down learning curve, results becoming more 

credible 

> Don’t let first cost estimate be final cost estimate 
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Closing Thoughts (2 of 2) 



• Concept design phases provide unique venue to facilitate 
exploring & converging system level design 

> Use the opportunity in these phases well 

> Not used well , the work of these phases usually will have to be re- 
done 

□ The later this realization occurs, the more expensive the resulting 
redesign is likely to be 
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a) Intermediate Systems Acquisition Course, Volume 2 Technical, Defense 
Acquisition University, Oct - Dec 1998, p. SE-1-24 and p. LM1-15 

b) Fundamentals of Systems Engineering (5 th Ed., day 3, chart 44), Strategy 
Bridge International, Inc., Academy of Program / Project & Engineering 
Leadership, presented 11-15 Feb 2013 at NASA / GSFC 

c) The NASA Mission Design Process, An Engineering Guide to the 
Conceptual Design, Mission Analysis and Definition Phases; the NASA 
Engineering Management Council; 22 Dec 1992 

d) GSFC-STD-1000F (with Administrative Changes), Rules for the Design 
Development, Verification, and Operation of Flight Systems 

e) Goddard Procedural Requirements 7120. 7, Schedule Margins and Budget 
Reserves to be Used In Planning Flight Projects and In Tracking Their 

Performance 
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Backup 
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Documenting Concept Design Results 
in Reports at End of Each Design Cycle 


• Provides official study record of what team did, how team 
did it, & what team found for present (& future) team use 

• Reports are developed for each subsystem / discipline 

> Built from standardized templates 

□ Include analysis methods & example calculations 

> Provide coherent technical waypoints that enable team to recall 
designs & performance from prior cycles, often needed for scaling or 

comparison 

□ High rate of design changes makes recollection difficult otherwise 

> Used for system level review, subsystem integration, independent 
review, new / follow-on team member orientation 

• Once approved , reports typically are under informal 
configuration control of Mission Systems Engineer 

> Briefings can be generated quickly from approved reports 

> Briefings contain only information in approved reports 
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Effect of Rounding Errors on Margin 
Determination 



• Rounding errors can significantly affect margin 
determination if adequate care isn’t exercised 

> In some cases, rounding errors can fully mask margins such as those 
for mass & power shown in Fig. 2 
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Effect of Rounding Errors on Margin 
Determination (Cont’d) 

Example 


Case 1: Power Available 

= 200 W 

Max. Estimated Power Required 

= 249 W 

Power Margin = 100 (200 W-249W)/ 249 W 

= -19.7% 

Case 2: Power Available 

= 200 W 

Max. Estimated Power Required 

= 151 W 

Power Margin = 100 (200 W- 151 W) / 151 W 

= 32.5 % 


The margins for Cases 1 and 2 are - 19.7 % and +32.5%, respectively 

Now consider a third case in which a designer rounds calculations to the first 
digit in Cases 1 and 2 

Case 3: Power Available = 2 x 10 2 W 

Max. Estimated Power Required = 2 x 10 2 W 

Power Margin = 100 (2 x 10 2 W- 2 x 10 2 W) / 2 x 10 2 W = 0% 


The margin for Case 3 is 0 % 
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Effect of Rounding Errors on Margin 
Determination (Cont’d) 

Example 


• Required power margin at end of pre-Phase A is 30% (Fig. 2) 

> Comparing Case 3 to Case 2 shows how rounding to 1st digit can 
fully mask a margin of over 30% 

> Additional errors can accrue when combinations of rounded results 
are used in successive calculations 


• To avoid masking resource margins , bookkeep design & 
performance calculations to 3 significant digits & report out 
to 2 significant digits 


• Notes: 



> This should not be taken to imply there is 3-digit accuracy in concept 
design work - there usually is not 

> This practice is simply a numerical safeguard to avoid propagating 
rounding errors that could overwhelm ability to adequately determine 
design or performance margins 

> Margin calculation method is per ref. (d), Table 1.06 
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Recognize Typical (but Unofficial) Phases 
of Concept Design 


• Concept design teams developing new designs that advance 
state of the art often experience four phases of work 

• 1) Unbridled Optimism 

> This phase features unbridled, optimistic performance desires levied 
as “ requirements ” before team gains credible understanding of 
associated cost & schedule 

> Meetings often not well-focused on study objectives 

□ Often feature unproductive, run-on advocacy discussions of why mission 
has best science of all competing missions & why it has best chance to 

win 

• 2) Shock 

> This brief phase usually begins after team completes its first credible 
cost estimate 
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Recognize Typical (but Unofficial) Phases 
of Concept Design (Cont’d) 


• 3) Denial 

> This phase features abundant rationalizations as to why models used 
to estimate costs weren’t representative 

> Team points to anything but excessively high technical capability as 
reason costs are too high in order that science return remains 
compelling relative to competition 

□ Seeks to reduce costs in areas other than technical capability / science 
return below normal allocations 

□ Theorizes why partner no-cost contributions will be higher than initially 

planned 

□ Argues why the request for proposal is incorrect, etc. 



• 4) Acceptance 

> This phase features the ultimate realization technical capability/ 
science return must be lowered to design a credible mission concept 

□ One that meets cost & schedule constraints according to established 
independent review standards 
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